System and method for enhanced collaborative forecasting

ABSTRACT

Systems and methods for amplifying the collective intelligence of networked human groups engaged in collaborative forecasting of future events having two possible outcomes. During a real-time session a computing device for each user displays a forecasting prompt and a dynamic user interface which includes a user-manipulatable marker moved by the user between a first limit and a second limit, where the position of the marker defines a forecasted probability of each possible outcome. The display also includes a first reward value and a second reward value, both of which are interactively responsive to the marker position. During a first time period, users manipulate the markers. After the first time period, a perturbation stimulus is displayed. During a second time period, users again manipulate the markers. After the second time period, a final group forecast is calculated based on the data collected during the first and second time periods.

This application is a continuation of U.S. application Ser. No.17/024,474 entitled SYSTEM AND METHOD OF NON-LINEAR PROBABILISTICFORECASTING TO FOSTER AMPLIFIED COLLECTIVE INTELLIGENCE OF NETWORKEDHUMAN GROUPS filed Sep. 17, 2020, which is a continuation of U.S.application Ser. No. 16/356,777 entitled NON-LINEAR PROBABILISTICWAGERING FOR AMPLIFIED COLLECTIVE INTELLIGENCE filed Mar. 18, 2019, nowU.S. Pat. No. 10,817,159, which claims the benefit of U.S. ProvisionalApplication No. 62/648,424 entitled NON-LINEAR PROBABILISTIC WAGERINGFOR AMPLIFIED COLLECTIVE INTELLIGENCE filed Mar. 27, 2018, which is acontinuation-in-part of U.S. application Ser. No. 16/230,759 entitledMETHOD AND SYSTEM FOR A PARALLEL DISTRIBUTED HYPER-SWARM FOR AMPLIFYINGHUMAN INTELLIGENCE, filed Dec. 21, 2018, now U.S. Pat. No. 10,817,158,claiming the benefit of U.S. Provisional Application No. 62/611,756entitled METHOD AND SYSTEM FOR A PARALLEL DISTRIBUTED HYPER-SWARM FORAMPLIFYING HUMAN INTELLIGENCE, filed Dec. 29, 2017, which is acontinuation-in-part of U.S. application Ser. No. 16/154,613 entitledINTERACTIVE BEHAVIORAL POLLING AND MACHINE LEARNING FOR AMPLIFICATION OFGROUP INTELLIGENCE, filed Oct. 8, 2018, now U.S. Pat. No. 11,269,502,claiming the benefit of U.S. Provisional Application No. 62/569,909entitled INTERACTIVE BEHAVIORAL POLLING AND MACHINE LEARNING FORAMPLIFICATION OF GROUP INTELLIGENCE, filed Oct. 9, 2017, which is acontinuation-in-part of U.S. application Ser. No. 16/059,698 entitledADAPTIVE POPULATION OPTIMIZATION FOR AMPLIFYING THE INTELLIGENCE OFCROWDS AND SWARMS, filed Aug. 9, 2018, now U.S. Pat. No. 11,151,460,claiming the benefit of U.S. Provisional Application No. 62/544,861,entitled ADAPTIVE OUTLIER ANALYSIS FOR AMPLIFYING THE INTELLIGENCE OFCROWDS AND SWARMS, filed Aug. 13, 2017 and of U.S. ProvisionalApplication No. 62/552,968 entitled SYSTEM AND METHOD FOR OPTIMIZING THEPOPULATION USED BY CROWDS AND SWARMS FOR AMPLIFIED EMERGENTINTELLIGENCE, filed Aug. 31, 2017, which is a continuation-in-part ofU.S. application Ser. No. 15/922,453 entitled PARALLELIZED SUB-FACTORAGGREGATION IN REAL-TIME SWARM-BASED COLLECTIVE INTELLIGENCE SYSTEMS,filed Mar. 15, 2018, claiming the benefit of U.S. ProvisionalApplication No. 62/473,424 entitled PARALLELIZED SUB-FACTOR AGGREGATIONIN A REAL-TIME COLLABORATIVE INTELLIGENCE SYSTEMS filed Mar. 19, 2017,which in turn is a continuation-in-part of U.S. application Ser. No.15/904,239 entitled METHODS AND SYSTEMS FOR COLLABORATIVE CONTROL OF AREMOTE VEHICLE, filed Feb. 23, 2018, now U.S. Pat. No. 10,416,666,claiming the benefit of U.S. Provisional Application No. 62/463,657entitled METHODS AND SYSTEMS FOR COLLABORATIVE CONTROL OF A ROBOTICMOBILE FIRST-PERSON STREAMING CAMERA SOURCE, filed Feb. 26, 2017 andalso claiming the benefit of U.S. Provisional Application No. 62/473,429entitled METHODS AND SYSTEMS FOR COLLABORATIVE CONTROL OF A ROBOTICMOBILE FIRST-PERSON STREAMING CAMERA SOURCE, filed Mar. 19, 2017, whichis a continuation-in-part of U.S. application Ser. No. 15/898,468entitled ADAPTIVE CONFIDENCE CALIBRATION FOR REAL-TIME SWARMINTELLIGENCE SYSTEMS, filed Feb. 17, 2018, now U.S. Pat. No. 10,712,929,claiming the benefit of U.S. Provisional Application No. 62/460,861entitled ARTIFICIAL SWARM INTELLIGENCE WITH ADAPTIVE CONFIDENCECALIBRATION, filed Feb. 19, 2017 and also claiming the benefit of U.S.Provisional Application No. 62/473,442 entitled ARTIFICIAL SWARMINTELLIGENCE WITH ADAPTIVE CONFIDENCE CALIBRATION, filed Mar. 19, 2017,which is a continuation-in-part of U.S. application Ser. No. 15/815,579entitled SYSTEMS AND METHODS FOR HYBRID SWARM INTELLIGENCE, filed Nov.16, 2017, now U.S. Pat. No. 10,439,836, claiming the benefit of U.S.Provisional Application No. 62/423,402 entitled SYSTEM AND METHOD FORHYBRID SWARM INTELLIGENCE filed Nov. 17, 2016, which is acontinuation-in-part of U.S. application Ser. No. 15/640,145 entitledMETHODS AND SYSTEMS FOR MODIFYING USER INFLUENCE DURING A COLLABORATIVESESSION OF REAL-TIME COLLABORATIVE INTELLIGENCE, filed Jun. 30, 2017,now U.S. Pat. No. 10,353,551, claiming the benefit of U.S. ProvisionalApplication No. 62/358,026 entitled METHODS AND SYSTEMS FOR AMPLIFYINGTHE INTELLIGENCE OF A HUMAN-BASED ARTIFICIAL SWARM INTELLIGENCE filedJul. 3, 2016, which is a continuation-in-part of U.S. application Ser.No. 15/241,340 entitled METHODS FOR ANALYZING DECISIONS MADE BYREAL-TIME COLLECTIVE INTELLIGENCE SYSTEMS, filed Aug. 19, 2016, now U.S.Pat. No. 10,222,961, claiming the benefit of U.S. ProvisionalApplication No. 62/207,234 entitled METHODS FOR ANALYZING THE DECISIONSMADE BY REAL-TIME COLLECTIVE INTELLIGENCE SYSTEMS filed Aug. 19, 2015,which is a continuation-in-part of U.S. application Ser. No. 15/199,990entitled METHODS AND SYSTEMS FOR ENABLING A CREDIT ECONOMY IN AREAL-TIME COLLABORATIVE INTELLIGENCE, filed Jul. 1, 2016, claiming thebenefit of U.S. Provisional Application No. 62/187,470 entitled METHODSAND SYSTEMS FOR ENABLING A CREDIT ECONOMY IN A REAL-TIME SYNCHRONOUSCOLLABORATIVE SYSTEM filed Jul. 1, 2015, which is a continuation-in-partof U.S. application Ser. No. 15/086,034 entitled SYSTEM AND METHOD FORMODERATING REAL-TIME CLOSED-LOOP COLLABORATIVE DECISIONS ON MOBILEDEVICES, filed Mar. 30, 2016, now U.S. Pat. No. 10,310,802, claiming thebenefit of U.S. Provisional Application No. 62/140,032 entitled SYSTEMAND METHOD FOR MODERATING A REAL-TIME CLOSED-LOOP COLLABORATIVE APPROVALFROM A GROUP OF MOBILE USERS filed Mar. 30, 2015, which is acontinuation-in-part of U.S. patent application Ser. No. 15/052,876,filed Feb. 25, 2016, entitled DYNAMIC SYSTEMS FOR OPTIMIZATION OFREAL-TIME COLLABORATIVE INTELLIGENCE, now U.S. Pat. No. 10,110,664,claiming the benefit of U.S. Provisional Application No. 62/120,618entitled APPLICATION OF DYNAMIC RESTORING FORCES TO OPTIMIZE GROUPINTELLIGENCE IN REAL-TIME SOCIAL SWARMS, filed Feb. 25, 2015, which is acontinuation-in-part of U.S. application Ser. No. 15/047,522 entitledSYSTEMS AND METHODS FOR COLLABORATIVE SYNCHRONOUS IMAGE SELECTION, filedFeb. 18, 2016, now U.S. Pat. No. 10,133,460, which in turn claims thebenefit of U.S. Provisional Application No. 62/117,808 entitled SYSTEMAND METHODS FOR COLLABORATIVE SYNCHRONOUS IMAGE SELECTION, filed Feb.18, 2015, which is a continuation-in-part of U.S. application Ser. No.15/017,424 entitled ITERATIVE SUGGESTION MODES FOR REAL-TIMECOLLABORATIVE INTELLIGENCE SYSTEMS, filed Feb. 5, 2016 which in turnclaims the benefit of U.S. Provisional Application No. 62/113,393entitled SYSTEMS AND METHODS FOR ENABLING SYNCHRONOUS COLLABORATIVECREATIVITY AND DECISION MAKING, filed Feb. 7, 2015, which is acontinuation-in-part of U.S. application Ser. No. 14/925,837 entitledMULTI-PHASE MULTI-GROUP SELECTION METHODS FOR REAL-TIME COLLABORATIVEINTELLIGENCE SYSTEMS, filed Oct. 28, 2015, now U.S. Pat. No. 10,551,999,which in turn claims the benefit of U.S. Provisional Application No.62/069,360 entitled SYSTEMS AND METHODS FOR ENABLING AND MODERATING AMASSIVELY-PARALLEL REAL-TIME SYNCHRONOUS COLLABORATIVESUPER-INTELLIGENCE, filed Oct. 28, 2014, which is a continuation-in-partof U.S. application Ser. No. 14/920,819 entitled SUGGESTION ANDBACKGROUND MODES FOR REAL-TIME COLLABORATIVE INTELLIGENCE SYSTEMS, filedOct. 22, 2015, now U.S. Pat. No. 10,277,645, which in turn claims thebenefit of U.S. Provisional Application No. 62/067,505 entitled SYSTEMAND METHODS FOR MODERATING REAL-TIME COLLABORATIVE DECISIONS OVER ADISTRIBUTED NETWORKS, filed Oct. 23, 2014, which is acontinuation-in-part of U.S. application Ser. No. 14/859,035 entitledSYSTEMS AND METHODS FOR ASSESSMENT AND OPTIMIZATION OF REAL-TIMECOLLABORATIVE INTELLIGENCE SYSTEMS, filed Sep. 18, 2015, now U.S. Pat.No. 10,122,775, which in turns claims the benefit of U.S. ProvisionalApplication No. 62/066,718 entitled SYSTEM AND METHOD FOR MODERATING ANDOPTIMIZING REAL-TIME SWARM INTELLIGENCES, filed Oct. 21, 2014, which isa continuation-in-part of U.S. patent application Ser. No. 14/738,768entitled INTUITIVE INTERFACES FOR REAL-TIME COLLABORATIVE INTELLIGENCE,filed Jun. 12, 2015, now U.S. Pat. No. 9,940,006, which in turn claimsthe benefit of U.S. Provisional Application 62/012,403 entitledINTUITIVE INTERFACE FOR REAL-TIME COLLABORATIVE CONTROL, filed Jun. 15,2014, which is a continuation-in-part of U.S. application Ser. No.14/708,038 entitled MULTI-GROUP METHODS AND SYSTEMS FOR REAL-TIMEMULTI-TIER COLLABORATIVE INTELLIGENCE, filed May 8, 2015, which in turnclaims the benefit of U.S. Provisional Application 61/991,505 entitledMETHODS AND SYSTEM FOR MULTI-TIER COLLABORATIVE INTELLIGENCE, filed May10, 2014, which is a continuation-in-part of U.S. patent applicationSer. No. 14/668,970 entitled METHODS AND SYSTEMS FOR REAL-TIMECOLLABORATIVE INTELLIGENCE, filed Mar. 25, 2015, now U.S. Pat. No.9,959,028, which in turn claims the benefit of U.S. ProvisionalApplication 61/970,885 entitled METHOD AND SYSTEM FOR ENABLING AGROUPWISE COLLABORATIVE CONSCIOUSNESS, filed Mar. 26, 2014, all of whichare incorporated in their entirety herein by reference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates generally to systems and methods forcollective intelligence, and more specifically to systems and methodsfor enabling users to provide consistent assessments.

2. Discussion of the Related Art

When capturing the insights from the members of a population it is oftenimportant to record not just their expressed sentiment, but also thestrength of their sentiment, whether it be their level of confidence ina predicted outcome or their level of conviction in an expressed pointof view. For example, when asking the members of a population to makeforecasts about future events, it is often valuable to record not justtheir forecasted outcome, but their confidence in that forecastedoutcome.

Soliciting honest expressions of confidence is particularly importantwhen aggregating sentiment across a population, as some members mayharbor a sentiment with only slight confidence, while other members mayharbor very strong confidence. In such situations, it is sometimesdesirable to aggregate sentiments across uses such that they areweighted by expressed confidence (or conviction) of each member.

The question remains—how can we best capture confidence or convictionlevels from participants?

In the past we have used various methods to address this need. Forexample, we have required participants to assign numerical probabilities(0% to 100%) that reflect their expected likelihood of an outcome.Alternatively, we have required participants to place wagers onoutcomes, the level of their wagers reflecting their confidence orconviction. Whatever the method we choose, the objective is—to getparticipants to (a) report their confidence (or conviction) asaccurately as they can, and (b) do so using a numerical scale that ishighly consistent from person to person.

Unfortunately, current methods often perform poorly on both fronts.That's because human participants have been shown to be inaccurate atreporting their internal confidence (or conviction) on numerical scales.This is true when asked to express confidence as percentages or wagers.

In addition, current methods are often inconsistent from person toperson. For example, when asking participants to predict which team willwin a sporting event—Team A or Team B, along with a confidence value, isdifficult to get accurate and consistent confidence values. Why is it sodifficult to get accurate and consistent confidence values? The problemsfaced include:

(a) Participants need to be motivated to give an accurate assessment oftheir personal confidence in a prediction they have expressed. If theyare not sufficiently motivated, they will not thoughtfully considertheir feelings.

(b) Participants need to express their confidence in an authentic mannerthat evokes a genuine visceral feeling of confidence vs uncertaintyrather than being asked to report a value on an abstract psychometricscale (like a number from 1 to 10, or a percentage from 0 to 100.) Priorresearch shows that abstract psychometric scales generate results whichare highly variable from participant to participant, while authenticexpressions are superior.

One approach that aims to motivate participants in an authentic manner,is to ask participants to place a wager upon their chosen outcome. Forexample, if they predict Team A will win, we can then ask how much theywould bet on that outcome. If they stand to benefit from a correctprediction in proportion to their wager, they may express confidence inan authentic manner.

We have used this method in the past, through a value we call “DollarConfidence”. While our results for Dollar Confidence have enabledamplification of intelligence, analysis shows that this value is notideal. The fact is, people are inconsistent when asked to place wagers,as personality differences cause some individuals to be risk adverse,and other individuals to be risk tolerant.

So, how can we drive participants to give authentic, accurate, andconsistent expressions of confidence when making forecasts? Aninnovative solution is needed that provides participants with a new formof wagering, that forces them to think probabilistically and reduces thedifferences between risk adverse and risk tolerant personalities.

SUMMARY OF THE INVENTION

Several embodiments of the invention advantageously address the needsabove as well as other needs by providing an interactive system foreliciting from a user a probabilistic indication of the likelihood ofeach of two possible outcomes of a future event, the interactive systemcomprising: a processor connected to graphical display and a userinterface; a graphical user interface presented upon the graphicaldisplay and including a user manipulatable wager marker that can bemoved by the user across a range of positions between a first limit anda second limit, wherein the first limit is associated with a firstoutcome of the two possible outcomes and the second limit is associatedwith a second outcome of the two possible outcomes; a first reward valuepresented on the graphical display and visually associated with thefirst outcome, the first reward value interactively responsive to theposition of the user manipulatable wager marker; a second reward valuepresented on the graphical display and visually associated with thesecond outcome, the second reward value interactively responsive to theposition of the user manipulatable wager marker; a first softwareroutine configured to run on the processor, the first software routineconfigured to repeatedly update both the first and second reward valuesin response to user manipulation of the wager marker, the first softwareroutine using a non-linear model for updating the first and secondreward values in response to linear manipulation of the wager marker,the non-linear model following a monotonic power function that isimplemented such that a non-linear increase in the first reward valuecorresponds to a non-linear decrease of the second reward value, and anon-linear increase in the second reward value corresponds to anon-linear decrease in the first reward value; a second software routineconfigured to run on the processor and determine a final first rewardvalue and a final second reward value based upon a final position of thewager marker; and a third software routine configured to run on theprocessor and generate a forecast probability value associated with eachof the two possible outcomes based upon the final position of the wagermarker between the two limits, the forecast probability value being alinear function of the position of the wager marker.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features and advantages of severalembodiments of the present invention will be more apparent from thefollowing more particular description thereof, presented in conjunctionwith the following drawings.

FIG. 1 is a schematic diagram of an exemplary computing deviceconfigured for use in a collaboration system.

FIG. 2 is a schematic diagram of an exemplary real-time collaborationsystem.

FIG. 3 is a flowchart of an exemplary group collaboration process usingthe collaboration system.

FIG. 4 is a flowchart of a method for interactive behavioral polling inaccordance with some embodiments of the present invention.

FIG. 5 is an exemplary user interface at a first point in time during anexemplary interactive behavioral polling session in accordance with someembodiments of the present invention.

FIG. 6 is an exemplary user interface at a second point in time duringthe exemplary interactive behavioral polling session in accordance withsome embodiments of the present invention.

FIG. 7 is an exemplary user interface at a third point in time duringthe exemplary interactive behavioral polling session in accordance withsome embodiments of the present invention.

FIG. 8 is an exemplary user interface at a fourth point in time duringthe exemplary interactive behavioral polling session in accordance withsome embodiments of the present invention.

FIG. 9 is an exemplary user interface at a fifth point in time duringthe exemplary interactive behavioral polling session in accordance withsome embodiments of the present invention.

FIG. 10 is a flowchart for training a Machine Learning algorithm usingbehavioral data in accordance with one embodiment of the presentinvention.

FIG. 11 is an exemplary slider interface display with a slider at aneutral position, in accordance with one embodiment of the presentinvention.

FIG. 12 is the exemplary slider interface display of FIG. 11 with theslider in a second position.

FIG. 13 is the exemplary slider interface display of FIG. 11 with theslider in a third position.

FIG. 14 is the exemplary slider interface display of FIG. 11 with theslider in a fourth position.

FIG. 15 is a flowchart of a wagering method in accordance with anotherembodiment of the present invention.

FIG. 16 is an exemplary slider interface display with a slider in aneutral position, in accordance with another embodiment of the presentinvention.

FIG. 17 is the exemplary slider interface display of FIG. 16 with theslider in a first position.

FIG. 18 is a flowchart of a wagering method including expressed andimplied probabilities in another embodiment of the present invention.

Corresponding reference characters indicate corresponding componentsthroughout the several views of the drawings. Skilled artisans willappreciate that elements in the figures are illustrated for simplicityand clarity and have not necessarily been drawn to scale. For example,the dimensions of some of the elements in the figures may be exaggeratedrelative to other elements to help to improve understanding of variousembodiments of the present invention. Also, common but well-understoodelements that are useful or necessary in a commercially feasibleembodiment are often not depicted in order to facilitate a lessobstructed view of these various embodiments of the present invention.

DETAILED DESCRIPTION

The following description is not to be taken in a limiting sense, but ismade merely for the purpose of describing the general principles ofexemplary embodiments. Reference throughout this specification to “oneembodiment,” “an embodiment,” or similar language means that aparticular feature, structure, or characteristic described in connectionwith the embodiment is included in at least one embodiment of thepresent invention. Thus, appearances of the phrases “in one embodiment,”“in an embodiment,” and similar language throughout this specificationmay, but do not necessarily, all refer to the same embodiment.

Furthermore, the described features, structures, or characteristics ofthe invention may be combined in any suitable manner in one or moreembodiments. In the following description, numerous specific details areprovided, such as examples of programming, software modules, userselections, network transactions, database queries, database structures,hardware modules, hardware circuits, hardware chips, etc., to provide athorough understanding of embodiments of the invention. One skilled inthe relevant art will recognize, however, that the invention can bepracticed without one or more of the specific details, or with othermethods, components, materials, and so forth. In other instances,well-known structures, materials, or operations are not shown ordescribed in detail to avoid obscuring aspects of the invention.

Real-time occurrences as referenced herein are those that aresubstantially current within the context of human perception andreaction.

As referred to in this specification, “media items” refers to video,audio, streaming and any combination thereof. In addition, the audiosubsystem is envisioned to optionally include features such as graphicequalization, volume, balance, fading, base and treble controls,surround sound emulation, and noise reduction. One skilled in therelevant art will appreciate that the above cited list of file formatsis not intended to be all inclusive.

Historical research demonstrates that the insights generated by groupscan be more accurate than the insights generated by individuals in manysituations. A classic example is estimating the number of beans in ajar. Many researchers have shown that taking the statistical average ofestimates made by many individuals will yield an answer that is moreaccurate than the typical member of the population queried. When theindividuals provide their input as isolated data points, to beaggregated statistically, the process is often referred to as“crowdsourcing”. When the individuals form a real-time system andprovide their input together, with feedback loops enabling the group toconverge on a solution in synchrony, the process is often referred to as“swarming”. While very different entities, crowds and swarms both shareone characteristic—the thoughts, feelings, insights, and intuitions ofhuman participants need to be captured and represented as data in anaccurate manner that can be processed across trials and acrosspopulations. Inconsistencies in the data captured from participants,because of difficulties in expressing their internal thinking asexternal reports can significantly degrade the ability to amplify theintelligence of a population. Swarms outperform crowds because theycapture data indicating behaviors of participants, but the requirementthat all members of a swarm participate at the same time, is alogistical constraint. What is needed is a method that combines theasynchronous benefits of polling (i.e. the participants do not need toall be engaged at the same time) and the behavioral benefits of swarming(i.e. that participant don't just report, they interact over time,revealing far more about their internal thinking than they might even beable to consciously express). The present invention addresses this bycreate a new form of interactive behavioral polling, combined withmachine learning, to optimize the collective intelligence and/orcollective insights of populations.

Interactive Behavioral Polling and Machine Learning

Human participants possess deep insights across a vast range of topics,having knowledge, wisdom, and intuition that can be harnessed andcombined to build an emergent collective intelligence. A significantproblem, however, is that people are very bad at reporting thesentiments inside their heads and are even worse at expressing theirrelative levels of confidence and/or conviction in those sentiments.Reports from human participants are inconsistent from trial to trial,using scales that are highly non-linear, and highly inconsistent fromparticipant to participant. To solve this problem, innovative methodsand systems have been developed to change the process of poll-based“reporting” to a dynamic process of “behaving” such that behavioral datais collected and processed which gives far deeper insights into the truesentiments of human participants, as well as far deeper insights intothe confidence and/or conviction that go with the expressed sentiments.

The methods and systems involve providing a question prompt, providing adynamic interface with data that is tracked and stored over time, andproviding a countdown timer that puts temporal pressure on theparticipant to drive their behavior.

The methods and systems further involve then providing the perturbationprompt which inspires the participant to adjust their answer, whileagain providing the dynamic interface with data that is tracked andstored over time, and providing the countdown timer that puts temporalpressure on the participant to drive their behavior. The perturbationprompt is an authoritative reference point which may be a real orfictional indication of an alternative view on the answer in manypreferred embodiments. The usefulness of a fictional indication is thatthe direction and magnitude of the perturbation (with respect to theparticipant's initial answer) can be varied across the pre-plannedspectrum across trials and across the population, so as to capture aspectrum of behavioral data for processing and machine learning.

Referring next to FIG. 4, a flowchart of a method for interactivebehavioral polling in accordance with some embodiments of the presentinvention is shown. In one embodiment, the method is implemented with anembodiment of the distributed architecture described in the relatedapplications (generally, with each user interacting with the computingdevice 100 and group computations performed by the CCS 142, although itwill be understood that other suitable computing and networking systemsmay be used to implement the methods described herein). Dynamic userinterfaces for an example of the interactive behavioral polling areshown in FIGS. 5-8.

Generally, Human participants possess deep insights across a vast rangeof topics, having knowledge, wisdom, and intuition that can be harnessesand combined to build a collective intelligence. A significant problem,however, people are very bad at reporting the sentiments inside theirheads and are even worse at expressing their relative levels ofconfidence and/or conviction in those sentiments. Reports from humanparticipants are inconsistent from trial to trial, using scales that arehighly non-linear, and highly inconsistent from participant toparticipant. To solve this problem, these innovative methods and systemshave been developed to change the process of poll-based “reporting” to adynamic process of “behaving” such that behavioral data is collected andprocessed which gives far deeper insights into the true sentiments ofhuman participants, as well as far deeper insights into the confidenceand/or conviction that go with the expressed sentiments.

The interactive behavioral polling method involves providing a prompt toeach user in the group, providing a dynamic interface with data that istracked and stored over time, and providing a countdown timer that putstemporal pressure on each participant to drive their behavior.

The methods and systems further involve then providing a perturbationprompt which inspires each participant to adjust their answer, whileagain providing dynamic interface with data that is tracked and storedover time, and providing a countdown timer that puts temporal pressureon the participant to drive their behavior. The perturbation prompt isan authoritative reference point which may be a real or fictionalindication of an alternative view on the answer in many preferredembodiments. The usefulness of a fictional indication is that thedirection and magnitude of the perturbation (with respect to theparticipants initial answer) can be varied across the pre-plannedspectrum across trials and across the population, so as to capture aspectrum of behavioral data for processing and machine learning.

A good way to describe the systems and methods enabled by the hardwareand software disclosed herein is by example. In the example illustratedin FIGS. 5-9, the example is a sports prediction, although the samemethods and systems can be used for financial predictions, politicalpredictions, and other types of prediction. The same methods and systemscan also be used for capturing and amplifying the accuracy of subjectivesentiments, such as views and opinions of a population.

In this example, the objective is to predict the outcome of footballgames, forecasting both the winner of the game and the number of pointsthe winner wins by. In this example, a set of 15 games will be forecastby a population of 100 human participants, with the objective ofaggregating the data from the 100 human participants to generate themost accurate group forecast possible. Traditionally, this would be doneby asking a single question and capturing static data, which has all theproblems described above. In the inventive system and method, a dynamicprocess is provided using a unique interface, unique prompts, and uniquetimer, and a unique machine learning process.

FIGS. 5-9 show an exemplary user interface for a single user in thegroup for one forecasted game at various points in time during theinteractive behavioral polling method of FIG. 4. This example is asports prediction, although the same methods and systems can be used forfinancial predictions, political predictions, and other types ofprediction. The same methods and systems can also be used for capturingand amplifying the accuracy of subjective sentiments, such as views andopinions of a population. In this example, the objective is to predictfootball games, forecasting both the winner of the game and the numberof points the winner wins by. In this example, a set of 15 games will beforecast by a population of 100 human participants, with the objectiveof aggregating the data from the 100 human participants to generate themost accurate group forecast possible. Traditionally, this would be doneby asking a single question and capturing static data, which has all theproblems described above. In the inventive system and method, a dynamicprocess is provided using a unique interface, unique prompts, and uniquetimer, and a unique machine learning process.

In a first provide prompt and dynamic user interface step 400 of theinteractive behavioral polling method, the CCS 142 sends to eachcomputing device 100 instructions and data for displaying the prompt andthe dynamic user interface. A dynamic user interface is presented toeach user. As illustrated in FIG. 5, an exemplary user interface 500 isshown during the first step 400. The dynamic user interface 500 includesa selection line 502, a first choice 504 located at a left-hand end ofthe selection line 502, a second choice 506 at a right-hand end of theselection line 502, a plurality of selection values 508, each associatedwith a point on the selection line 502, and a moveable slider 510. InFIG. 5 the slider 510 is shown at an initial origin location on theselection line 502, but the slider 510 may initially be shown at anylocation on the selection line 502. Also shown is a countdown timer 512and the prompt 514.

In this example, the first choice 504 is displayed as “49ers” and thesecond choice 506 is displayed as “Raiders”. The selection values 508range from +16 on the 49ers (left) side to +16 on the Raiders (right)side of the selection line 502. An origin is located at the center ofthe selection line 502. The prompt 514 is displayed as “Who will win andby how much, the San Francisco 49ers or the Oakland Raiders?”.

It will be understood by those of ordinary skill in the art thatalthough the exemplary interface is a slider, the dynamic datacollecting and prompting methods can be achieved with a variety of otherlayouts.

As illustrated in the exemplary display 500, in this case the prompt 514is a textual question, and the interface is a sliding-type interface.The prompt 514 will appear on the screen under software control of thecomputing device 100, allowing control of the exact timing of when theuser reads the question and provides a response. In preferredembodiments, during step 402, the countdown timer 512 also initiallyappears. The countdown of the time will give time pressure to the user.As shown in FIG. 5 below, the slider 510 is moveable by the user alongthe selection line 502 and enables the user to choose which team willwin, and by how many points, by positing the slider 510 to the left orright by a given amount. This can be done by a mouse or touchscreeninterface, or by gesture interface. The display 500 may also include a“done” button 516, which would, during the time period, end the timeperiod even if there is still time left in the time period.

In the next provide dynamic input step 404, during a pre-determined timeperiod the user is allowed to provide input to indicate his response tothe prompt. If the countdown timer 512 is shown, the countdown timer 512counts down during the time period. During the dynamic input step 404,the computing device 100 tracks the user input, collecting data aboutthe timing, position, speed, acceleration, and trajectory of the user'sresponse using the dynamic interface (in this case the slider 512). Anytime delay between the prompt 514 appearing on the display and a firstmotion of the slider 512 is captured as well. This behavioralinformation reflects not just a reported final answer, but alsoindicates the internal confidence and/or conviction the participant hasin the expressed value, especially when processed by machine learning inlater steps. For example, a slower movement of the slider may result inthe user being assigned a lower confidence value.

An example of the dynamic interface 500 at the end of the first timeperiod is shown in FIG. 6. In this example, the user has moved theslider 510 to a location on the left-hand side of the selection line502, indicating a selection of the 49ers to win by 9 points.

In some embodiments, a textual indication 600 of the selection locationmay be shown. FIG. 6 also shows the countdown timer 512 reading “0:00”,thus indicating to the user that the input time period has ended.

The system generally includes the countdown timer 512, so the user feelspressured to act promptly, but still has significant time—for example,15 seconds counting down in the example above. The user might move theslider 510 quickly during the 15 seconds, or they might adjust andreadjust, all of which is captured by the software. The system alsogenerally includes the clickable or pressable “DONE” button (as shown inFIG. 5), indicating (before the time period is up) that the user hasfinished, allowing them to move on, even if the counter has not yetgotten to zero.

Again, the important thing is that the software tracks behavioralinformation for the user, not just the final resting position of theslider 510. This behavioral information includes (1) the time delaybetween the prompt and the user first grabbing the slider and moving it,(2) the speed of the slider, (3) the motion dynamics of the slider—doesit go straight to an answer or does it overshoot and come back or doesit get adjusted and readjusted during the allotted time period, as thecountdown timer ticks down.

In the next perturbation analysis step 406, after the first time periodhas ended or the user clicks the “done” button 516, the software is thenconfigured to determine (and then present to the user) a perturbationstimulus, which will drive a second round of behavior to be captured inreal-time during a second time period. This may be an indicator tellingthem what an “EXPERT” thinks the answer is to the given prompt. This maybe an indicator telling them what an “AI” thinks the answer is to thegiven prompt. This may be an indicator telling them what a “population”thinks the answer is to the given prompt. The primary goal ispsychological—to inform them that a voice of authority and/orcredibility has an alternate view to be considered. In many preferredembodiments, the perturbation is fictional. In other embodimentsperturbation can be real (based on actual expert or AI or population).The population may be the group of users currently engaged with thebehavioral polling method through individual computing devices 100 andthe perturbation indicator may be derived based on a statistical mean ormedian of their initial input values prior to any perturbation.

In some preferred embodiments, the perturbation analysis step 406includes collecting and processing user dynamic input data from aplurality of users responding to the prompt, each interacting with acomputing device connected to the server, such that the perturbationstimulus is generated based at least in part on the data from theplurality of users responding to the prompt. In this way, a statisticalmean or median or other statistical aggregation can be used alone or inpart when generating the perturbation stimulus.

In some embodiments the perturbation indicator communicated to each ofthe plurality of computing devices is identical, such that allparticipants are given the same perturbation. In other embodiments, adistribution of perturbations is calculated and transmitted such that arange of perturbation indicators are communicated to the plurality ofcomputing devices. In some such embodiments the distribution is a normaldistribution. In some embodiments, the distribution is a randomizeddistribution. An inventive aspect of using a distribution is that thepopulation of participants are provided with a range of uniqueperturbation stimuli and thereby provide a range of responses to saidstimuli for analysis and processing.

In some embodiments the perturbation indicator computed and provided toa first group of participants is based on data collected from a secondgroup of participants.

For this particular example embodiment, the perturbation is a fictionalperturbation where the system displays a selection and identifies it asan “expert opinion”. The perturbation may be randomly selected to beeither higher or lower than the user's response, by a random margin. Or,instead of a random margin, a pre-planned distribution of margins acrossall users may be employed. For example, if 100 people were given thissurvey, a pre-planned distribution of margins (above or below the user'sinitial prediction) may be used for the expert perturbation, enablingthe system to collect a diverse data set that has a range of desiredperturbations. Alternately, a range of perturbations could be given notacross all users who are answering this same question, but across eachgiven user, across a set of predictions (for example, across a set of 10games being predicted).

In the present perturbation stimulus step 408, the perturbation stimulusis shown on the display. An example of the display of the perturbationstimulus is shown in FIG. 7, where a perturbation stimulus 700 isdisplayed as an arrow located at the +6 point on the left-hand side,with the accompanying text “Experts=+6”, indicating to the user an“expert” selection of the 49ers winning by 6 points.

Next, in the provide prompt for updating step 410, The system isconfigured to prompt the user again, giving them the opportunity duringa second time period to update their prediction. In the next second timeperiod step 412, during the second time period the user has the optionto adjust the location of the slider 510. In a preferred embodiment, acountdown timer again appears during the second time period, giving theuser, for example, 15 seconds to update their prediction. The user couldjust click the “DONE” button 516 during the second period and not adjustthe location of the slider 510. Or the user could adjust the location ofthe slider 510. During the second time period the software tracks notjust the final answer, but again tracks the behavioral dynamics of theprocess—including (1) the time delay between the prompt and the usergrabbing the slider 510 and (2) the speed of the slider motion, and (3)the trajectory of the slider motion, and (4) how quickly the usersettles on an updated location. This additional behavioral data, duringthe second time period in response to the perturbation stimulus, is afurther indicator of confidence/conviction in the answer given,especially when processed by machine learning.

FIG. 8 illustrates user input at a time during the second time period,where the user has adjusted the location of the slider 510 from theprevious location of 49ers+9 to an updated location of 49ers+7 (asindicated by the position of the slider 510 and the textual indication600). The updated position of the slider 510 for this particular userindicates that he or she tempered the personal prediction in response tothe perturbation. Some users may not be influenced at all. Again, inmany embodiments the prompt is fictional, although the user would notknow that, for that allows the software to inject a fixed distributionof perturbation prompts across questions to a single user, and/or acrossthe full population of users. In other embodiments, the prompt is notfictional but computed based on authoritative data, for example from anexpert source or a computer model. In a particularly useful embodiment,the prompt is computed in real-time based on the initial data providedby the group of individuals engaging the behavioral polling system onthe same question. Thus, the group of users provide input prior to theperturbation. The central server 142 computes a statistical mean ormedian of the groups input. A perpetration prompt is then generated anddisplayed based at least in part upon the statistical median or medianof the group's input. In some embodiments, statistical variance isapplied around the statistical mean or median to ensure that thepopulation gets a range of values as stimuli, rather than all gettingthe same perturbation stimuli.

After the second time period is ended, in optional calculate scalingfactor step 414, the computing device 100 (or the CCS 142, if the datais sent over the network to the CCS 142) calculates a scaling factorbased on the data collected during the first and second time periods.This scaling factor could be then used when the user is participating ina real-time collaborative session, as previously described in therelated applications.

In the next calculate prediction step 416, the CCS 142 receives the datafrom each user computing device 100 participating in the pollingsession, and computes a predication related to the prompt using thebehavioral data collected from all users during the time periods, inaddition to other data. In the optional final display prediction step418, the CCS 142 sends an indication of the prediction to each computingdevice 100, and each computing device 100 displays the prediction on thedisplay, as illustrated in FIG. 9, with a prediction 900 of the 49erswinning by 4 points shown. This forecast is not a simple average of theinput reported from the 100 participants, but instead is a weightedaggregation, the weighting based at least in part on the behavioral datacollected from participants reflecting their relative confidence and/orconviction in their individual responses.

Behavioral data for each user for each polling session is stored. At thecompletion of all instances of the dynamic poll, across the set ofquestions (for example all 15 NFL football games being played during agiven week) and across a population (for example, 100 football fans), adetailed and expressive behavioral data set will have been collected andstored in the database of the system (for example, send to and stored inthe CCS 142)—indicating not just a set of final predictions across a setof users, but representing the confidence and/or conviction in thosepredictions, especially when processed by machine learning. In someembodiments the data obtained during the interactive behavioral pollingmethod can be used to identify from a group of users a sub-population ofpeople who are the most effective and/or insightful participants, to beused in a real-time swarming process.

Typical “Wisdom of Crowds” for making a sports prediction collects a setof data points for a particular game, wherein each data point wouldindicate each user's single forecast for which team will win, and by howmuch. If there were 100 users predicting the 49ers/Raiders game, theprocess would generate an average value across the simple data set andproduce a mean. The problem is, every user in that group has (a) adifferent level of confidence and/or conviction in their answer, (b)they are very poor at expressing or even knowing their confidence, and(c) if asked to report their confidence, every individual has a verydifferent internal scale—so confidence can't be averaged with anyaccuracy. Thus, traditional methods fail because they combinepredictions form a population of very different individuals, but thosevalues are not all equivalent in their scales, confidence or accuracy.

To solve this problem, the unique behavioral data collected in the stepsof FIG. 4 above, combined with unique machine learning techniques, hasenabled us to scale (and aggregate) the contributions of eachparticipant in the population based on a more accurate indication oftheir relative confidence in their personal predictions, and/or therelative accuracy of those predictions. An exemplary flowchart fortraining a Machine Learning algorithm using the behavioral data is shownin FIG. 10.

In a first obtain and store behavioral data step 1000, the systemcollects behavioral data using the interactive polling method aspreviously described. For example, the system collects behavioral datathat includes thinking delay time between when the prompt 514 appears onthe screen for a given user and that user starts to move the slider 510,in combination with elapsed time taken for the user to settle on aninitial answer using the slider 510, the max speed of the slider 510,and the total distance traveled of the slider 510 over time (includingovershoots and/or double-backs), along with the user's answer value forthe initial prompt phase, and the adjustment amount that the userchanged their answer when prompted with a perturbation, as well as theelapsed time taken for the adjustment, the max speed during theadjustment, and the total distance traveled of the slider 510 during theadjustment (including overshoots and/or double-backs).

In the next obtain additional data step 1002, additionalevent/user/group data is obtained. For example, in some embodiments, thesystem collects and/or computes, for each user, the amount of time theyspent pulling away from the majority sentiment in the population. Inaddition to the above behavioral data, the system also may also collectand store data regarding how correct each user was in their initialestimate, as well as how correct they were in their final estimate inthe face of the perturbance. In some embodiments, the users are requiredto also engage in a real-time swarm, where how much time the user spendspulling against the swarm (i.e. being defiant to the prevailingsentiment) is tracked.

In some embodiments, users are asked “How much would you bet on thisoutcome?” as a means of gathering further confidence data. In some suchembodiments, the dynamic behavioral data is used to estimate confidenceby training on the correlation between the behavioral data and the “Howmuch would you bet on this outcome?” answers. (This is called ‘dollarconfidence’ in our current methods).

In some embodiments, webcam data is collected from users during theinteractive period while they are dynamically manipulating the sliderinput in response to the prompt. This webcam data, which is configuredto capture facial expressions, is run through sentiment analysissoftware (i.e. facial emotion determination and/or sentimentdetermination and/or engagement determination) to generate and storesentiment, emotion, and/data engagement data from each user. Forexample, tools like Emotient® and/or EmoVu may be used, or othersuitable emotion detection API. Facial emotion, sentiment, and/orengagement data collected during the first time period, and separatelycollected during the second adjustment time period driven by theperturbation, are stored for users for each question they answer. Facialdata is thus a secondary form of behavioral data, indicating confidenceand/or conviction of users during their interactive responses. This datacan then be used in later steps to (a) optimize predictions ofconfidence in the answer, and/or (b) optimize predictions of accuracy inthe answer.

In the computer user score step 1004, the system computes a user scoreusing an algorithm and the obtained data. In one example, the systemcomputes a skill score for each user. Using a football game as anexample, if the true outcome of the game is +4, and the user's initialguess is +9, the user might get an initial accuracy score of: |4−9|=5,where the lower the score, the better, with a perfect score being 0. Iftheir updated score was +8, they will get an updated accuracy score of:|4−8|=4, where the lower the score, the better, with a perfect scorebeing 0. The updated accuracy score is also calculated withconsideration as to whether the perturbance was an influencer towardsthe correct score, or away from the correct score. In one suchembodiment, the updated accuracy score could be a function of both theuser's initial and final skill scores. As an example, if the user'sinitial guess is +9, and then after the perturbation gives a final guessof +7, where the true outcome of the game is +4, the skill score may becalculated as: |4−9|+|4−7|=8. These scores act as a measure of theuser's skill in prediction overall.

In other embodiments, instead of computing accuracy (i.e. how correctthe initial prediction and updated predictions were), the systemcomputes scores for user confidence and/or conviction. Defiance time canused by the current system to compute a defiance score. The defiancescore and/or other real-time behavioral data is of unique inventivevalue because it enables weighting of participants without needinghistorical performance data regarding the accuracy of prior forecasts.

In the final train machine learning algorithm step 1006, the behavioraldata and/or the user scores are used to train a machine learning system.For example, the system can train the machine learning algorithm on userconfidence and/or conviction scores. The system can train a machinelearning algorithm using the defiance score to estimate the defiancetime of each user. With this estimate, we are able to weight new users'contributions to new crowds based on their estimated defiance time,thereby increasing the accuracy of the new crowd.

By training a Machine Learning algorithm on these scores, the system canpredict which users are more likely to be most skillful at answering thequestion (e.g. generating an accurate forecast) and which users are lesslikely to do so. With this predicted skill level, the software of thesystem is then able to weight new users' contributions to new crowds (orswarms) based on their behavioral data and the combination of otherfactors, thereby increasing the accuracy of the new crowd (or swarm).

The dynamic behavioral data obtained can be used in an adaptive outlieranalysis, for example as previously described in related applicationSer. No. 16/059,658 for ADAPTIVE OUTLIER ANALYSIS FOR AMPLIFYING THEINTELLIGENCE OF CROWDS AND SWARMS. For example, the behavioral data canbe used in combination with other characteristics determined from thesurvey responses for use in machine learning, including the outlierindex for that user, as described in the aforementioned related patentapplication. The contribution of each user to that statistical averagecan be weighted by (1−Outlier_Index) for that user. Similarly, whenenabling users to participate in a real-time swarm, the User Intentvalues (as disclosed in the related applications) that are applied inreal time, can be scaled by a weighting factor of (1−Outlier_Index) forthat user. In this way, users are statistically most likely to provideincorrect insights are either removed from the population and/or havereduced influence on the outcome. What is significant about this methodis that it does not use any historical data about the accuracy ofparticipants in prior forecasting events. It enables a fresh pool ofparticipants to be curated into a population that will give amplifiedaccuracy in many cases.

In addition to using the behavioral data disclosed above to make a moreaccurate crowd prediction, as described above, it is also useful to usethe behavioral data to identify from the population, a sub-population ofpeople who are the most effective and/or insightful participants, to beused in a real-time swarming process. The present invention enables thecuration of human participants by using dynamic behavioral data in twoinventive forms—(I) taking an initial pool of baseline participants andculling that pool down to a final pool of curated participants which arethen used for crowd-based or swarm-based intelligence generation, and/or(II) taking a pool of baseline participants and assigning weightingfactors to those participants based on their likelihood of givingaccurate insights (i.e. giving a higher weight within a swarm toparticipants who are determined to be more likely to give accurate,correctly-confident insights than participants who are determined to beless likely to give accurate, over- or under-confident insights). Insome inventive embodiments, both culling and weighting are used incombination—giving a curated pool that has eliminated the participantswho are most likely to be low insight performers, and weighting theremaining members of the pool based on their likelihood of beingaccurate insight performers.

For example, the behavioral data described above, may be used in someembodiments, in combination with other characteristics determined fromthe survey responses for use in machine learning, including the OUTLIERINDEX for that user, as described in the aforementioned co-pendingpatent application Ser. No. 16/059,658.

The contribution of each user to a question in a crowd can be weightedby a machine-learned representation of their confidence and predictedaccuracy. Similarly, when enabling users to participate in a real-timeswarm, the User Intent values that are applied in real time can bescaled by a weighting factor that is machine-learned from the Outlierindex and behavioral data. In this way, users are statistically mostlikely to provide consistently incorrect insights are either removedfrom the population and/or have reduced influence on the outcome. Whatis significant about this method is that it does not use any historicaldata about the accuracy of participants in prior forecasting events. Itenables a fresh pool of participants to be curated into a populationthat will give amplified accuracy in many cases. (This amplification isbased only on analysis of that users responses and behaviors to thecurrent set of questions, which does not require historical accuracydata for that user).

In some embodiments of the present invention, a plurality of values aregenerated for each participant within the population of participantsthat reflect that participant's overall character across the set ofevents being predicted. outlier index is one such multi-event value thatcharacterizes each participant with respect to the other participantswithin the population across a set of events being predicted. Inaddition, a confidence INDEX is generated in some embodiments of thepresent invention as a normalized aggregation of the confidence valuesprovided in conjunction with each prediction within the set ofpredictions. For example, in the sample set of questions provided above,each prediction includes Confidence Question on a scale of 0% to 100%.For each user, the confidence index is the average confidence the userreports across the full set of predictions, divided by the averageconfidence across all users across all predictions in the set. Thismakes the confidence index a normalized confidence value that can becompared across users. In addition, multi-event self-assessment valuesare also collected at the end of a session, after a participant hasprovided a full set of predictions.

In some embodiments of the present invention, a plurality of multi-eventcharacterization values are computed during the data collection andanalysis process including (1) Outlier Index, (2) Confidence Index, (3)Predicted Self Accuracy, (4) Predicted Group Accuracy, (5)Self-Assessment of Knowledge, (6) Group-Estimation of Knowledge, (7)Behavioral Accuracy Prediction, and (8) Behavioral ConfidencePrediction. In such embodiments, additional methods are added to thecuration step wherein Machine Learning is used to find a correlationbetween the multi-event characterization values and the performance ofparticipants when predicting events similar to the set of events.

In such embodiments, a training phase is employed using machine learningtechniques such as regression analysis and/or classification analysisemploying one or more learning algorithms. The training phrase isemployed by first engaging a large group of participants (for example500 to 1000 participants) who are employed to make predictions across alarge set of events (for example, 20 to 40 baseball games). For each ofthese 500 to 1000 participants, and across the set of 20 to 40 events tobe predicted, a set of values are computed including an Outlier Index(OI) and at least one or more of a Confidence Index (CI), a PredictedSelf Accuracy (PSA), a Predicted Group Accuracy (PGA), a Self-Assessmentof Knowledge (SAK), a Group Estimation of Knowledge (GAK), a BehavioralAccuracy Prediction (BAP), and a Behavioral Confidence Prediction (BCP).

In addition, user performance data is collected after the predictedevents have transpired (for example, after the 20 to 40 baseball gameshave been played). This data is then used to generate a score for eachof the large pool of participants, the score being an indication of howmany (or what percent) of the predicted events were forecast correctlyby each user. This value is preferably computed as a normalized valuewith respect to the mean score and standard deviation of scores earnedacross the large pool of participants. This normalized value is referredto as a Normalized Event Prediction Score (NEPS). It should be notedthat in some embodiments, instead of discrete event predictions, userpredictions can be collected as probability percentages provided by theuser to reflect the likelihood of each team winning the game, forexample in a Dodgers vs. Padres game, the user could be required toassign percentages such as 78% likelihood the Dodgers win, 22%likelihood the Padres win. In such embodiments, alternate scoringmethods may be employed by the software system disclosed here. Forexample, computing a Brier Score for each user or other similar costfunction.

The next step is the training phase wherein the machine learning systemis trained (for example, using a regression analysis algorithm or aneural network system) to find a correlation between a plurality of thecollected characterization values for a given user (i.e. a plurality ofthe Outlier Index, the Confidence Index, a Predicted Self Accuracy, aPredicted Group Accuracy, a Self-Assessment of Knowledge, a GroupEstimation of Knowledge, a Behavioral Accuracy Prediction, and aBehavioral Confidence Prediction) and the Normalized Event PredictionScore for a given user. This correlation, once derived, can then be usedby the inventive methods herein on characterization value data collectedfrom new users (new populations of users) to predict if the users arelikely to be a strong performer (i.e. have high normalized EventPrediction Scores). In such embodiments, the machine learning system(for example using multi-variant regression analysis) will provide acertainty metric as to whether or not a user with a particularcombination of characterization values (including an Outlier Index) islikely to be a strong or weak performer when making event predictions.In other embodiments, the machine learning system will select a group ofparticipants from the input pool of participants that are predicted toperform well in unison.

Thus, the final step in the Optimization and Machine Learning process isto use the correlation that comes out of the training phase of themachine learning system. Specifically, the trained model is used byproviding as input a set of characterization values for each member of anew population of users, and generating as output a statistical profilefor each member of the new population of users that predicts thelikelihood that each user will be a strong performer based only on theircharacterization values (not their historical performance). In someembodiments the output is rather a grouping of agents that is predictedto perform optimally. This is a significant value because it enables anew population of participants to be curated into a high performingsub-population even if historical data does not exist for those newparticipants.

Non-Linear Probabilistic Wagering

To solve the problem of obtaining authentic, accurate, and consistentexpressions of confidence for a user making a prediction, an innovativeapproach for soliciting confidence from human forecasters has beendeveloped. Rather than ask participants to assign probabilities to theirforecast (which is too abstract for most participants to accuratelyprovide), or asking participants to place simple wagers on outcomes(which is too susceptible to variations in risk tolerance), a newmethodology has been created where participants express the relativeprobability of each outcome, but do so in a way that is presented as anauthentic wager, and which motivates participants to be as accurate asthey can.

This solution is called Non-Linear Probabilistic Wagering (NPW). It is amethodology in which the participants (a) are asked to place a wager oneach outcome, thereby requiring them to make an authentic assessment oftheir confidence in each possible result, and (b) requires users todistribute capital between the possible outcomes, based not on a simplelinear scale as is used in traditional wagering, but using a novel MeanSquare Difference scale.

Furthermore, the innovation enables distribution of wagers in responseto an easily understood interface control, like a slider interface ordial interface. This allows participants to move an element (like aslider) and adjust the relative wagers on the possible outcomes of aforecasted event, the non-linear computations happening automatically.An example slider interface display 1100 including a slider selectionline 1104 with a slider 1102 (also referred to in general as a usermanipulatable wager marker) at a neutral (center) position is shown inFIG. 11.

It's important to note that the above exemplary slider interface display1100, while appearing simple, performs unlike any prior confidenceinterface that we know of. As will be described later in this document,the values assigned to each side of the slider selection line 1102 varywith slider position in a unique and powerful way.

Specifically, this method enables participants to place wagers upon thepredicted outcomes but does so using a unique non-linear scale thatmodels probabilistic forecast without the users needing to be skilled inthinking in terms of probabilities or even needing to know anythingabout probabilities.

The users just need to think in terms of wagers. In some embodiments theusers are authentically motivated, for example when their compensationis tied to the true outcome of these events. For example, the users onlywin wagered amounts (real winnings or simulated points) for the outcomethat actually happens in the real event.

This method can be described with respect to the sequence of sliderpositions shown in the FIGS. 11-14. FIGS. 11-14 represent a singleslider interface at different positions chosen by a user, wherein eachposition reflects different split wagers defined in software by thenovel mean-squared-difference scale. FIG. 11 shows a linear sliderinterface display 1100 with the slider 1102 at a neutral (center)position on the slider selection line 1104 between a first limit 1114and a second limit 1116. First limit 1114 is visually associated with afirst (left-hand) outcome 1108 and second limit 1116 is visuallyassociated with a second (right-hand) outcome 1110. FIG. 12 shows theslider interface display 1200 with the slider 1102 adjusted to aposition on the slider selection line 1104 closer (leftward) to thefirst limit 1114. FIG. 13 shows a slider interface display 1300 with theslider 1102 adjusted to a position on the slider selection line 1104even closer to the first limit 1114. FIG. 14 shows a slider interfacewith the slider 1102 adjusted to the first limit 1114.

Referring next to FIG. 15, a flowchart of an embodiment of the novelwagering method is described with reference to FIGS. 11-14.

In the present embodiment, the methods are executed by one or moresoftware routines configured to run on the processor of the computingdevice. In the initial display user interface step 1500, first, agraphical user interface is displayed, in this case the linear sliderinterface display 1100, with the amount paid for each outcome (firstreward value 1106 and second reward value 1110) clearly identified. Instep 1502 the user can adjust some aspect of this interface (in theexamples of FIGS. 11-14, the slider 1102 moved linearly along theselection line 1104 between the two outcomes) to demonstrate theirbelief in either outcome 1108, 1112, as shown in FIGS. 12-14, where theslider 1102 has been moved along the selection line 1104 towards thefirst outcome 1108. Each location on the slider interface display 1100,1200, 1300, 1400 has some associated probability (not shown to the user)of the leftmost event (in the exemplary display “Chelsea wins”) as p1,and the rightmost event (in the exemplary display as “Arsenal wins”) as1−p1. These probabilities are referred to as forecast probabilityvalues. The probabilities associated with each location must bemonotonic (probability only increasing as the slider approaches thetarget), but they do not have to be linear. In some embodiments themonotonic power function is a mean-squared function. In step 1504, theprobabilities for the current slider location is determined. In step1506, the new outcome reward values 1106, 1110 are calculated based onthe slider 1102 location as:

Pay if Chelsea Wins=100*(1−(1−p1)²)

Pay if Arsenal Wins=100*(1−(p1)²)

In this way the reward values 1108, 1110 are interactively responsive tothe position of the slider 1102. In some embodiments, such as shown inFIGS. 11-14, the first and second reward values are dollar amounts.

In step 1508 the reward values 1106, 1110 are updated in the display1100, 1200, 1300, 1400 in real time to reflect the interface's currentpay-outs.

In the next decision step 1510, the user decides whether he is satisfiedwith his wager. If the user is satisfied, the method proceeds to step1512 and the user submits the wager with the slider at the currentvalue. If the user is not satisfied with the wager, the method returnsto step 1502, and modifies the slider location. In this way the user caninteract with the interface until they are satisfied with the wagersplit and submit their wager (for example, the wagers shown in FIGS.11-14 could be different interactions of the user during the session).In step 1514, the probabilities (one for each outcome) that define thesubmitted wager are recorded as the Expressed Probability (also referredto as the forecast probability values). The Expressed Probability is theprobabilities that the user placed on the outcome through the wageringinterface.

In some embodiments, the system comprises a first a first softwareroutine configured to run on the processor, the first software routineconfigured to repeatedly update a first and a second reward value inresponse to user manipulation of a wager marker between a first limitand a second limit, the first software routine using a non-linear modelfor updating the first and second reward values in response to linearmanipulation of the wager marker, the non-linear model following amonotonic power function that is implemented such that a non-linearincrease in the first reward value corresponds to a non-linear decreaseof the second reward value, and a non-linear increase in the secondreward value corresponds to a non-linear decrease in the first rewardvalue.

In some embodiments the system comprises a second software routineconfigured to run on the processor and determine a final first rewardvalue and a final second reward value based upon a final position of thewager marker.

In some embodiments the system comprises a third software routineconfigured to run on the processor and generate a forecast probabilityvalue associated with each of the two possible outcomes based upon thefinal position of the wager marker between the two limits, the forecastprobability value being a linear function of the position of the wagermarker.

In some embodiments the system comprises a scoring software routineconfigured to run on the processor, the scoring software routineconfigured to be executed after an actual outcome of the future event isknown, the scoring software routine configured to assign a score to theuser based at least in part upon the final reward value associated withthe actual outcome.

In some embodiments the Expressed Probability is then mapped to anImplied Probability, which represents the real-world probability of theevent after accounting for human biases and individual risk aversion.

Another inventive aspect of this methodology is the mapping of Expressedto Implied probabilities, which can be computed based on either (a)behavioral and/or performance history of a general pool of participants,(b) the behavioral and/or history of this user, or (c) a combination ofa and b.

For example, one mapping can be found using a technique called DrivenSurveys. In this novel technique, individuals interact with theProbabilistic Wagering program over a series of survey questions and arepaid a bonus depending on their wagering success. The questions have aknown probabilistic outcome, which the users are told about, and thenthey are asked to distribute wagers using the unique slider systemabove. In this way, we get a direct mapping between probabilities thatare known to the users and the wager splits that they produce fromauthentic visceral response.

In the example shown in FIGS. 16 and 17, in an example question in aDriven Survey, a participant may be told that they are flipping a loadedcoin. This particular coin has an 80% chance of turning up heads, and a20% chance of turning up tails. Through the Probabilistic Wageringsystem described above, they can assign their dollar amounts to bothheads and tails in whatever proportion they believe is best for theirpotential gains. An example slider interface display 1600 with the firstoutcome 1108 as “heads” and the second outcome 1112 as “tails” is shownbelow in FIG. 16 with the slider 1102 in the starting (center) positionon the selection line 1104.

But again, the odds in this example were “driven” such that the userknows there is an 80% change of Heads turning up, and a 20% chance oftails. The user proceeds to adjust the position of the slider 1102according to the method of FIG. 15. How the user adjusts the slider 1102position is recorded, with the reward values 1106, 1110 numbers varyingbased on the equations above. In this particular case, users willgenerally give results that deviate from the true probabilisticequation, enabling us to then calibrate using a heuristic, a linearregression, or a machine learning model. The slider 1102 position forideal mapping is shown in the slider interface display 1700 of FIG. 17.

As you can see, the dollar amounts for reward values 1106 and 1110 inFIG. 17 are not obvious in their relation to 80%, which is the power ofthis method—it hides the probabilistic nature of the forecast, evokingpurely visceral responses, which can then be calibrated through amapping from Expressed Probability to Implied Probability.

After the survey is complete and the results of the simulated coin flipsknown, they receive pay (or points) in proportion to the wagers theymade. This motivates users to place their wagers in the proportion theybelieve will maximize their expected return, rather than splitting theirmoney evenly or placing it all on one side.

This process allows the Implied Probability (the wager that the usermade) to be associated with the true probability of an event (thelikelihood of a coin flipping heads) for any individual who takes thistest. Additionally, it allows generalizations to be made about themapping for an average person by taking the statistical average over allusers who have taken the test.

This unique method can be described by the flowchart of FIG. 18. Steps1800 through 1814 operate the same as the analogous steps in theflowchart of FIG. 15. In step 1814 (analogous to step 1514 of FIG. 15),the Expressed Probability is recorded. The method then proceeds to step1516, where the expressed probability value based on the slider locationis transformed into the Implied Probability. Using a historicalknowledge of the user or of users in general, the system transforms theexpressed probability into an implied real-world probability of theevent. This can be normalized for varying levels of human bias andindividual risk-aversion in a measurement.

Referring again to FIG. 18, in some embodiments the system comprises anadditional bias calibration software routing configured to run on theprocessor and take as input the forecast probability values (theexpressed probability) and generate as output a calibrated forecastvalue (implied real-world probability value).

In some embodiment the bias calibration routine uses an optimizedmapping from the forecast probability values to the calibrated forecastvalue, where the mapping is generated using historical data captured fora population of users who have previously used the interactive system,the historical data including forecast probability values and knownoutcomes for a set of prior events.

In some embodiments the bias calibration routine uses an optimizedmapping from forecast probability values to the calibrated forecastvalue, where the mapping is generated using historical data captured forthe user during a series of previous uses of the interactive system, thehistorical data including forecast probability values and known outcomesfor a set of prior events.

One thing that both poll-based methods and swarm-based methods have incommon when making predictions based on input from populations ofparticipants, is that a smarter population generally results in moreaccurate forecasts. As disclosed in co-pending U.S. patent applicationSer. No. 16/059,698 by the current inventors, entitled “ADAPTIVEPOPULATION OPTIMIZATION FOR AMPLIFYING THE INTELLIGENCE OF CROWDS ANDSWARMS” which is hereby incorporated by reference, methods and systemsare disclosed that enables the use of polling data to curate a refinedpopulation of people to form a swarm intelligence. While this method iseffective, by incorporating improved assessments of participantconfidence using the unique NPW process above, deeper and more accurateassessments of human confidence and human conviction are attained andused to significantly improve the population curation process.Specifically, this enables higher accuracy when distinguishing membersof the population of are likely to be high-insight performers on a givenprediction task as compared to members of the population who are likelyto be low-insight performers on a given prediction task and does sowithout using historical data about their performance on similar tasks.Instead we can perform outlier analysis to determine which forecasts theparticipant went against the convention wisdom, and then look at theirNPW confidence to determine if they were self-aware that their pickswere perceived as risky by the general population.

While many embodiments are described herein, it is appreciated that thisinvention can have a range of variations that practice the same basicmethods and achieve the novel collaborative capabilities that have beendisclosed above. Many of the functional units described in thisspecification have been labeled as modules, in order to moreparticularly emphasize their implementation independence. For example, amodule may be implemented as a hardware circuit comprising custom VLSIcircuits or gate arrays, off-the-shelf semiconductors such as logicchips, transistors, or other discrete components. A module may also beimplemented in programmable hardware devices such as field programmablegate arrays, programmable array logic, programmable logic devices or thelike.

Modules may also be implemented in software for execution by varioustypes of processors. An identified module of executable code may, forinstance, comprise one or more physical or logical blocks of computerinstructions that may, for instance, be organized as an object,procedure, or function. Nevertheless, the executables of an identifiedmodule need not be physically located together, but may comprisedisparate instructions stored in different locations which, when joinedlogically together, comprise the module and achieve the stated purposefor the module.

Indeed, a module of executable code could be a single instruction, ormany instructions, and may even be distributed over several differentcode segments, among different programs, and across several memorydevices. Similarly, operational data may be identified and illustratedherein within modules, and may be embodied in any suitable form andorganized within any suitable type of data structure. The operationaldata may be collected as a single data set, or may be distributed overdifferent locations including over different storage devices, and mayexist, at least partially, merely as electronic signals on a system ornetwork.

While the invention herein disclosed has been described by means ofspecific embodiments, examples and applications thereof, numerousmodifications and variations could be made thereto by those skilled inthe art without departing from the scope of the invention set forth inthe claims.

What is claimed is:
 1. A system for forecasting a future event byaggregating input from a group of users, the system comprising: aplurality of computing devices, each including a graphical display, auser interface, a forecasting application running on the computingdevice, and configured for network communication, wherein each computingdevice is associated with one member of the group of users; a server innetworked communication with each of the plurality of computing devicesand including a server application running on the server, wherein thesystem is configured to: display, on each of the computing devices, aforecasting prompt and a dynamic user interface for capturing anassociated user's dynamic response to the forecasting prompt, whereinthe dynamic user interface includes: a user-manipulatable marker thatcan be moved by the associated user to input a forecasting valueresponsive to the forecasting prompt, provide a first forecasting periodduring which the associated user of each of the plurality of computingdevices independently adjusts the user-manipulatable marker to set aninitial forecasting value responsive to the forecasting prompt, saidinitial forecasting value collected and stored by the server from eachof the plurality of computing devices as provided by each member of thegroup of users; after the first forecasting period has ended, display aperturbation stimulus on each of the plurality of computing devices, theperturbation stimulus based at least in part on data collected by theserver from the plurality of computing devices during the firstforecasting period; provide a second forecasting period following thedisplay of the perturbation stimulus wherein the second forecastingperiod starts at substantially the same time for all members, duringwhich the associated user of each of the plurality of computing devicesadjusts their user-manipulatable marker to set an updated forecastingvalue responsive to the forecasting prompt, said updated forecastingvalue collected and stored by the server from each of the plurality ofcomputing devices as provided by each member of the group of users;calculate an adjustment amount for each of the plurality of computingdevices, the adjustment amount for each computing device indicating achange between the initial forecasting value and the updated forecastingvalue collected from that computing device in response to theperturbation stimulus; and after the second forecasting period hasended, calculate a final group forecast by aggregating data from theplurality of the computing devices, said calculating based at least inpart on the adjustment amount computed for each of the plurality of thecomputing devices.
 2. The system of claim 1 wherein the perturbationstimulus is based at least in part on the initial forecasting valuescollected from the plurality of computing devices.
 3. The system ofclaim 1 wherein a different perturbation stimulus is provided to each ofthe plurality of computing devices, thereby providing a variety ofperturbation stimuli across the group of users.
 4. The system of claim 1wherein the forecasting value is a probability.
 5. The system of claim 1wherein the forecasting value reflects a predicted score of a futuresporting event.
 6. The system of claim 1 wherein the calculating of theadjustment amount is based at least in part on a time delay between theperturbation stimulus and a detected adjustment of theuser-manipulatable marker for each of at least a plurality of theplurality of computing devices.